Measuring utilization of resources in datacenters

ABSTRACT

For measuring component utilization in a computing system, a server energy utilization reading of a statistical significant number of servers out of a total number of servers located in the datacenter is obtained by measuring, at predetermined intervals, a collective energy consumed by all processing components within each server. The collective energy is measured by virtually probing thereby monitoring an energy consumption of individual ones of all the processing components to each collect an individual energy utilization reading, where the individual energy utilization reading is aggregated over a predetermined time period to collect an energy consumption pattern associated with the server utilization reading.

CROSS-REFERENCE TO RELATED APPLICATIONS

This Application is related to the following five Applications havingU.S. application Ser. Nos. 15/289,272, 15/289,274, 15/289,276,15/289,280, and 15/289,284, each filed on even date as the presentApplication.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates generally to large scale computing, andmore particularly measuring computing component utilization in adistributed computing environment.

Description of the Related Art

A popular type of large scale computing is cloud computing, in whichresources may interact and/or be accessed via a communications system,such as a computer network. Resources may be software-renderedsimulations and/or emulations of computing devices, storage devices,applications, and/or other computer-related devices and/or services runon one or more computing devices, such as a server. For example, aplurality of servers may communicate and/or share information that mayexpand and/or contract across servers depending on an amount ofprocessing power, storage space, and/or other computing resources neededto accomplish requested tasks. The word “cloud” alludes to thecloud-shaped appearance of a diagram of interconnectivity betweencomputing devices, computer networks, and/or other computer relateddevices that interact in such an arrangement.

Cloud computing may be provided as a service over the Internet, such asin the form of “Infrastructure as a Service” (IaaS), “Platform as aService” (PaaS), and/or “Software as a Service” (SaaS). IaaS maytypically provide physical or virtual computing devices and/oraccessories on a fee-for-service basis and onto which clients/users mayload and/or install, and manage, platforms, applications, and/or data.PaaS may deliver a computing platform and solution stack as a service,such as, for example, a software development platform, applicationservices, such as team collaboration, web service integration, databaseintegration, and/or developer community facilitation. SaaS may deploysoftware licensing as an application to customers for use as a serviceon demand. SaaS software vendors may host the application on their ownclouds or download such applications from clouds to cloud clients,disabling the applications after use or after an on-demand contractexpires.

The provision of such services allows a user access to as much in theway of computing resources as the user may need without purchasingand/or maintaining the infrastructure, such as hardware and/or software,that would be required to provide the services. For example, a user mayinstead obtain access via subscription, purchase, and/or otherwisesecuring access. Thus, cloud computing may be a cost effective way todeliver information technology services. A fundamental need exists toenhance the underlying systems and infrastructure which support andmaintain this fast-growing industry.

SUMMARY OF THE INVENTION

Various embodiments for measuring component utilization in a computingsystem, by a processor device, are provided. In one embodiment, a methodcomprises obtaining a server energy utilization reading of a statisticalsignificant number of servers out of a total number of servers locatedin the datacenter by measuring, at predetermined intervals, a collectiveenergy consumed by all processing components within each server; andmeasuring the collective energy by virtually probing thereby monitoringan energy consumption of individual ones of all the processingcomponents to each collect an individual energy utilization reading;wherein the individual energy utilization reading is aggregated over apredetermined time period to collect an energy consumption patternassociated with the server utilization reading.

In addition to the foregoing exemplary embodiment, various other systemand computer program product embodiments are provided and supply relatedadvantages. The foregoing Summary has been provided to introduce aselection of concepts in a simplified form that are further describedbelow in the Detailed Description. This Summary is not intended toidentify key features or essential features of the claimed subjectmatter, nor is it intended to be used as an aid in determining the scopeof the claimed subject matter. The claimed subject matter is not limitedto implementations that solve any or all disadvantages noted in thebackground.

BRIEF DESCRIPTION OF THE DRAWINGS

In order that the advantages of the invention will be readilyunderstood, a more particular description of the invention brieflydescribed above will be rendered by reference to specific embodimentsthat are illustrated in the appended drawings. Understanding that thesedrawings depict only typical embodiments of the invention and are nottherefore to be considered to be limiting of its scope, the inventionwill be described and explained with additional specificity and detailthrough the use of the accompanying drawings, in which:

FIG. 1 is a block diagram illustrating a hardware structure of adisaggregated computing environment, in which aspects of the presentinvention may be realized;

FIG. 2 is an additional block diagram illustrating a hardware structureof a disaggregated computing environment, in which aspects of thepresent invention may be realized;

FIG. 3 is a flowchart diagram illustrating a method for power managementin a disaggregated computing system, in accordance with aspects of thepresent invention;

FIG. 4 is block diagram illustrating a high level view of powermanagement in a disaggregated computing system, in accordance withaspects of the present invention;

FIG. 5 is a flowchart diagram illustrating a power management algorithmfor power management in a disaggregated computing system, in accordancewith aspects of the present invention; and

FIG. 6 is a chart diagram illustrating a priority schedule for powermanagement in a disaggregated computing system, in accordance withaspects of the present invention.

DETAILED DESCRIPTION OF THE DRAWINGS

Computing resources are usually pre-configured by vendors at fixedlevels of configurations. One aspect is that each individual computingresource, such as memory size, number of CPUs, disk size, etc. has alimited boundary. Another aspect is that each computing platform has alimited number of physical customization options. Today's workloads arerunning under these limitations, which subsequently is a reason thattechniques such as memory swapping and caching optimization are used incomputing environments.

The emergence of cloud computing changes the paradigm of how peopleutilize computing resources by providing a pay-as-you-go model. Thepublic cloud has been created by service providers to allow access tothose who need such computing resources on demand. As aforementioned,access to cloud resources is provided through the Internet or privatenetwork connections or through co-location of fixed infrastructure heldas a base, augmented by on demand resources when needed. The underlyinginfrastructure, however, is a set of fixed computing configurationswhich provide inflexibility when scaling or descaling demands areappropriate.

The underlying architecture of the Infrastructure as a Service (IaaS)cloud is generally traditional hardware used in data centers asdescribed above. Users either access the hardware directly, or accessvirtual machines contained thereon. However, because of the fixed natureof building servers as enclosures that are configured once, when theenclosure is built, the fundamental architecture underneath the datacenter is very rigid and inflexible. It is thus the cloud software thatprovides the emulation to create the flexible, on-demand functionalitythat cloud services are known for. This functionality is quite limitedhowever, as many mechanisms depend on software relying on serverenclosures, which architectures originated early in the PersonalComputer era, turning into an on-demand service.

The Virtual Machine (VM) is a software technique based on an entity thatruns on a part of a server, possibly with other such entities sharingthe same server. It represents the unit of on-demand computation, whereeach such entity is designated with a pre-defined number of virtual CPUsand memory. Once defined, a VM cannot change its base resources, such asadding memory or adding virtual CPUs. This is because such a VM sharesthe hardware resources of a fixed pre-built server enclosure with otherVMs, and it may not be possible to displace other users to make room forthe resource expansion of the first user. While such is possible inprinciple (e.g. by migrating other users (live VM migration) to otherservers), such an operation would create an abundant increase in trafficand require an overload on a datacenter network. In addition, theprovisioning of new VMs on-demand can take an impractical amount oftime, relatively speaking (e.g. minutes, while real-world events mayrequire a response to events in sub-second times). Thus the notion oftrue, real-world and corresponding on-demand cloud infrastructure doesnot exist. This situation may force users to provision resources forworse-case needs (max processor number/speed, max memory) and to keepVMs even if unneeded, only to be able to respond to real-world events inrelative time.

For cloud services achieved via Application Program Interfaces (APIs),users do not access the operating system directly, but rather issuerequests via the APIs. The computation is then handled by the underlyingoperating system and hardware infrastructure. Some vendors provide acertain level of scaling and elasticity that are transparent to userAPIs. However, the level of scaling is limited by the type ofapplication and by the capacity of the individual computing resource.For example, if a workload requires a high demand of memory usage, it isnot possible to scale up on memory size individually. Therefore, theoverall resource utilization is poor and this solution is notcost-effective either.

In view of the forgoing, disaggregated computing systems provideflexibility and elasticity in constructing bare-metal computing systemsfor use in the cloud, to provide on-demand flexibility to cloud users,or “tenants”. A disaggregated computing system is referred to as asystem with large pools of physical hardware resources, such as CPUs,accelerators, memory devices, and storage devices, whose connectivitywith each other individual hardware resource can be dynamically switchedwithout shutting down any hardware nor running applications. Individualhardware resources from these pools can be selected to assemble computersystems on-demand. Thus, a bare-metal computer system with a flexiblecapacity of individual computing resources may be assembled in adisaggregated system, such that workloads are computed based on hardwareresource configurations that are most suitable for the respectiveworkload. In one embodiment, for example, a system may be constructedwith an extremely high capability of memory size but with a moremoderate capacity of CPU and other resources, for a memory-intensiveworkload.

Within these disaggregated systems, various application-level servicelevel agreements (SLAs) may be employed to dynamically provision thehardware resources on-demand, and ensure that a tenant is receiving thecomputing service they have purchased, while retaining an overall costand performance efficiency model for both the cloud service provider andthe tenant.

Power Management in Disaggregated Systems

From an electrical power utilization perspective, modern data centeroperations have the following two desirable goals. First, in modern datacenters, it is desirable to run the resident systems as close to 100%component utilization as possible due to the large capital investment inthe servers, racks, cables, and storage, as well as the softwarelicensing costs integrated in such systems, etc. Therefore, poweringdown equipment (i.e. hibernate or sleep mode) is not desirable, as itwould mean all the invested capital of the hardware and softwarelicenses associated with such is wasted.

Second, given a range of electrical power allocated (contracted) fromthe utility company for a datacenter, it is highly desirable to operatewithin that allocated (contracted) power range with some small variance(e.g. +/−5% within the allocated power). Power is a valuable resourceand utility companies have limited total power they can generate andcarry over the power grid to supply. Utility companies cannot quicklyadjust the generation of power to match fast and large fluctuations ofpower consumed. Therefore, it is imperative that a certain range ofpower usage that is contracted, be consumed by the data center. Utilitycompanies need to balance electrical power generation with powerconsumption because their generators can adjust to periodic demands butcannot adjust for these large erratic power usage changes. When thedatacenter erratically underutilizes the contracted power, the utilitycompany may have to burn the extra power generated so that they do notdamage their generators. Consequently, utility power supply contractsmay stipulate that large variations in power usage by a customer (e.g.,a datacenter operator) may lead to costly penalties in form ofadditional charges. Hence, it is not always beneficial to quickly moveinto sleep mode and save power only to quickly move back to need to usethat power again.

There are significant performance, throughput, utilization, and costbenefits if the combination of the above two goals can be realized,where the beneficiary parties include the utility companies (costbenefits), data center/cloud provider (performance, utilization, andcost benefits), and end user/cloud customer (performance, throughput,and cost benefits). Realizing the combination of the two goals wouldhelp simultaneously increase utilization of hardware resources,performance and throughput of user workloads, while operating within theallocated power range, resulting in lower costs for the utility company,data center provider, and end user.

Based on the above two considerations, the present invention considersthe problem of managing electrical power allocation to the processingcores based on the SLAs needs of workloads they run. To address thisproblem, throughput of workloads must be optimized by dynamicallyre-adjusting the clock speeds and voltages at which processing cores(and hence, circuits, transistors clock frequency) operate. Throughputis different than performance, as it is possible to achieve higherperformance per thread or core but with much more power used relatively,versus using more cores with lower performance (i.e. lower clock speed)but less power per core to achieve higher overall throughput.

In a disaggregated computing environment, a customer (tenant) mayprovision a server with certain number of resources, of which not allmay require the same throughput (and thereby the same clock speed) torun workloads at all the times. For some workloads, it may be beneficialto re-adjust the voltage when requiring lower/higher clock speeds sothat power consumption is better aligned with the throughput at whichthe processing core operates. Note that the jump in higher performance(higher clock speed) is non-linear and hence, takes more power thanoperating at lower clock speeds. Though, if throughput is the goal, suchcan be achieved by lower clock speeds with more cores, however, whilethe overall power used is less, there is a higher use of resources (i.e.more processing cores).

State-of-the-art processors, servers and operating system have theability to adjust voltage and clock frequencies for each processor corebased on fixed policies such as a standardized fair share policy.However, the same cannot make such adjustments based on known, predictedor requested workload needs. Hence, they are unable to take SLA termsinto consideration when making changes to throughput of processingcores. To address such shortcomings, the present invention proposes atechnique to allocate power based on workloads' SLA commitments whilemaximizing the usage of total contracted power allocation from theutility company. Additionally, the present invention provides atechnique to determine total power for a datacenter that should becontracted from a utility company for the next contract term based onthe usage patterns and forecast.

The following terms and variables are used throughout the disclosure, ofwhich definitions are provided for clarity and understanding:

-   -   a. Total_Utilized: A total capability expressed in the terms of        electrical power currently utilized across all workloads in the        datacenter.    -   b. Total_Contracted: A total electrical power allocation        contracted from an electric utility company for the datacenter.    -   c. Contracted Variation: An electrical power range with the        acceptable variation as agreed upon between the datacenter and        the electric utility company (e.g. +/−5%) without incurring        penalties.    -   d. Total_Available: A total processing capability of the        datacenter expressed in the terms of electrical power which can        be re-allocated to workloads.        (Total_Available=Total_Contracted−Total_Utilized)    -   e. Workload_Current_Allocated: A current processing capability        expressed in the terms of electrical power allocated to each        workload.    -   f. Workload_Predicted_Demand: A resource usage prediction based        on a learned workload model.    -   g. Workload_Resource_Overage: A workload demand or surplus        capability expressed in the terms of electrical power based on        the predicted demand compared to the current resource        allocation.        (Workload_Resource_Overage=Workload_Current_Allocated−Workload_Predicted_Demand        for a particular workload, where if Workload_Resource_Overage>0,        then the workload has a surplus of processing capability        expressed in the terms of electrical power. Otherwise, the        workload requires more power than currently allocated).    -   h. Workload_Allocated_Proportion: A proportion of processing        capability expressed in the terms of electrical power that is        currently allocated to a specific workload versus the total        power allocated across all workloads.    -   i. Total_Workload_Allocated: A total processing capability        expressed in the terms of electrical power allocated across all        workloads.    -   j. Resource_Unit: A unit of resource being maximized. The        resource unit comprises the processing capability expressed in        the terms of electrical power (i.e. megawatts). Examples of        Resource_Units in this case may be additional processing        capability derived from increasing the power to processing cores        of a particular workload or processing components which can be        dynamically allocated to a particular workload.    -   k. Time_Window: A time window of a duration for which the        Total_Available electrical power is being maximized. Any        particular Time_Window may be defined with a configurable        granularity (e.g. per second, minute, hour, day, week etc.).

Turning now to FIG. 1, a block diagram of a disaggregated computingenvironment is illustrated, including cloud environment 100. Withincloud environment 100 is the disaggregated computing system comprisingphysical hardware resources 200. Physical hardware resources 200 maycomprise of classifications of the hardware resources such as a storagedevice pool 202, a Graphics Processing Unit (GPU) device pool 204, a CPUdevice pool 206, a memory device pool 208, and a network device pool210. The physical hardware resources 200 are in communication with apower management module 250. Power management module 250 may comprise ofsuch components as a workload resource utilization predictor 258, apower maximization engine 260, and a power allocator 262, each of whichmay be in communication with a workload statistics database 254, aconfiguration database 252, a workload resource utilization monitoringsystem 256, and a power provisioning engine 264.

As will be further discussed, the configuration database 252 storespredefined values of data center level variables (e.g. power expressedin Total_Contracted, Total_Utilized, and Total_Available). The workloadstatistics database 254 is responsible for receiving statisticalinformation, (such as for a unique workload ID, resources allocated(Workload_Current_Allocated), predicted demand(Workload_Predicted_Demand), resource overage(Workload_Resource_Overage), current consumption, average utilizationand other runtime statistics, workload SLA types, business logic andworkload impact) of the expected workloads in the datacenter.

The workload resource utilization predictor 258 models workload usagepatterns based upon metrics collected from the workload resourceutilization monitoring system 256, which is responsible for monitoringthe utilization of each workload in the system. The power allocator 262prepares power allocation instructions based on computations of thepower maximization engine 260 and sending the power allocationinstructions to the power provisioning engine 264. The powerprovisioning engine 264 invokes the instructions received from the powerallocator 262 (i.e. increases or decreases power to processing cores ofa particular workload or migrates freed-up processing components to aparticular workload). The power maximization engine 260 receives inputfrom the workload resource utilization predictor 258, and computes thevarious variables as previously defined.

In communication with the cloud environment 100, the power managementmodule 250, the provisioning component 260, and the physical hardwareresources 200, are tenants 212A, 212B, and 212 n. Tenants 212A, 212B,and 212 n may communicate with the cloud environment 100 provided by anysignal-bearing medium.

It should be noted that the elements illustrated in FIG. 1 provide onlyan example of related components that may be included in thedisaggregated computing architecture. For example, power managementmodule 250 may include other components than workload resourceutilization predictor 258, power maximization engine 260, powerallocator 262, workload statistics database 254, configuration database252, workload resource utilization monitoring system 256, and powerprovisioning engine 264, while staying in spirit and scope of thepresent invention. Additionally, the management module 250 and thecomponents therein may physically comprise of separate entities, or maybe combined into one entity. Furthermore, the duties of the powermanagement module 250, and thus the components therein, may be performedand comprised of physical components, computer code, or a combination ofsuch.

FIG. 2 is a block diagram illustrating the physical hardware resources200 portion of FIG. 1. Included in the storage device pool 202 arestorage devices 202A, 202B, and 202 n. The GPU device pool 204 includesGPU devices 204A, 204B, and 204 n. The CPU device pool 206 includes CPUdevices 206A, 206B, and 206 n. The memory device pool 208 includesmemory devices 208A, 208B, and 208 n. Finally, the network device pool210 includes network devices 210A, 210B, and 210 n. Each aforementionedhardware resource may be in communication with an additional one or moreaforementioned hardware resources via a signal-bearing medium.

Within physical hardware resources 200, each hardware resource appearingin solid line (i.e. storage device 202A, GPU device 204A, CPU device206A, memory device 208A, and network device 210A) are assigned hardwareresources to one or more tenants (i.e. tenants 212A, 212B, 212 n).Hardware resources appearing in dashed line (i.e. storage devices 202B,202 n, GPU devices 204B, 204 n, CPU devices 206B, 206 n, memory devices208B, 208 n, and network devices 210B, 210 n) are unassigned hardwareresources which are available on-demand for a respective tenant 212A-nworkload.

Each respective tenant 212A-n may be assigned individual respectivehardware resources 200 in arbitrary quantities. In one embodiment, eachrespective tenant 212A-n may be assigned an arbitrary quantity of anindividual respective hardware resource 200 within a limit of totalsystem capacity and/or an available quantity of the respective hardwareresources 200. For example, a memory device 208A-n allocated from thememory pool to a respective tenant 212A-n may be provided in a minimalunit of allocation (e.g. a byte or word) up to a limit of total systemcapacity and/or an available quantity of the memory devices 208A-n.

In another embodiment, each respective tenant 212A-n may be assignedindividual respective hardware resources 200 within a quantum stepsizing restriction. For example, memory devices 208A-n may need to beallocated on quantum sizes of full or half of memory DIMM units, toassure full bandwidth from the respective memory device 208A-n to theprocessor when reading/writing data. This is especially true in adisaggregated system since the memory device 208A-n is directlyconnected via fiber/optical switch to the processor memory unit (forread/write memory transactions) as if it was locally connected to theprocessor chip, but rather may be a small distance (e.g. 1 meter) awayin location. In another example, because the disaggregated system is notbased on virtual components but rather physical components (i.e. actualchips than cores or VMs), the quantum sizing restriction may requirethat a minimum of one CPU device 206A-n be assigned to a tenant 212A-n,with additional CPU devices 206A-n being provisioned to the tenant212A-n in two, four, etc. quantities.

In various described embodiments herein, the present invention usesexisting prediction techniques to estimate a workload's demand andallocates available power, or removes the workload's allocated power(based on a workload priority), which would be wasted, otherwise. Atotal available power is monitored and tracked. Power is maintained atas small variations as possible from the contracted utility power forthe datacenter. In some embodiments, the total power available isallocated to workloads in need based on the workload's priority toprovide them an additional throughput boost, while keeping the overallutilization within the range power contracted for such datacenter. Forexample, increasing the voltage of processing cores running such aprioritized workload and thereby increasing the clock speed will useadditional power, yet for a good cause (e.g. large data analytics thatcan be performed whenever higher priority workloads are not usingpower). If there is not sufficient total available power in thedatacenter to match maximum contracted utility power while the workloadsrequire additional power based on the predicted demand, the systemremoves the estimated surplus power and/or reduces the voltage allocatedto some of the workloads (with lower priority) and allocates it to thehigher prioritized workloads based on their predicted demand.

The priority of the workloads is determined based on their SLAs, orother business rules, as assigned to tenants 212 a-n. By allocating thesurplus power available and driving the total power available towardszero within the variation from what was nominally allocated, the systembalances the total power consumed at the datacenter and matches moreefficiently with the range of power that was actually contracted fromthe utility company.

The present invention therefore attempts to maximize the utilization ofpower contracted from utility companies rather than reducing powerconsumption, where SLAs or other business logic requirements are takeninto consideration when maximizing the use of the power allocated. Thisis an important distinction from prior art, which primarily hibernatesor powers down under-utilized servers to save power consumption. Inanother prior art, power is adjusted based on high/low usage but it doesnot focus on maximizing the use of the power allocated from the utilitycompany by leveraging workloads with opportunistic SLAs to adjust thepower delivery, while maintaining SLAs. Additionally, and at the sametime, the disclosed invention also improves the utilization andthroughput capacity of the deployed hardware and software at adatacenter.

As aforementioned, this functionality is achieved through a policy basedpower management service, which implements a technique to maintain thepower utilization variation within the range allocated (contracted) bythe utility company while maximizing such use of power and itsutilization thereby achieving the desired workload throughput. This isuniquely achievable in a disaggregated computing system as processingcomponents can be quickly switched from one workload to another andwhere large, disaggregated systems have the capability of driving manyconcurrent workloads. Hence, there are always workloads to be performedwhether active or in suspended mode, as well as other workloads to beresumed, all of which may be triggered quickly within milliseconds ofresponse time.

Using the aforementioned example, if a utility company has allocated(contracted) 10 megawatts of power with an acceptable ±5% variation withno penalties for such variation of power use, then the datacenter isallocated a power range of 9.5-10.5 megawatts. Even though the range isacceptable, a datacenter with high capital investment would prefer to beon upper bound of the total power utilization (i.e. over-utilize ratherthan under-utilize), thus driving the total additional available powerwithin the datacenter towards zero while maintaining a goal of executingthe allowed 10.5 megawatts.

One fundamental principle recognized by the functionality of the presentinvention is that energy/power used by each processor/processor core(not only in a disaggregated computing system but in general computing)is a measure of work and utilization of performing respective workloadsversus a maximum amount of power or energy each individualprocessor/processor core is able to consume over a given period of time.This maximum amount of power or energy able to be consumed by theindividual processor is dependent upon an individual specification(design) of the chip itself, the amount of heat the processor is able todissipate based upon its cooling budget, and an amount of availablepower (e.g. an amount of power available from the utility company, orinput power specifications). For example, as very-large scaleintegration (VLSI) chip technology advances, and smaller and smallertransistors can be fabricated within a given area of a chip, morefunctions are possible—however while these transistors can also switchfaster and perhaps operate at lower voltages than previous largertransistors, they may either not all be working (not all transistors ina chip will switch) and/or the energy or power available may remainconstant due to cooling limitations, power delivery designs, etc.

Stated differently, as is the current state of the art, processorutilization is generally calculated based on its operating system stateof all the processes the processors/processor cores have “run” versus“idle” processes running over a given period of time. However, when aprocess is in a “run” state, the amount of work it performs (i.e. theamount of power or energy it consumes) depends heavily on the coderunning by the processor. For example, some codes are heavy in floatingpoints while others are performing memory transactions and are stalling(for a short enough time that the operating system cannot measure) oncache misses while the memory is read (even when multi-threading/issuesare sent to the memory controllers from all the cores). Thus, an I/Ooperation may have a higher wait time such that if a core/process waitson an I/O, the process is usually marked as “I/O wait” by the operatingsystem. This does not provide an accurate measurement for the amount of“work” or otherwise a base utilization of the processor/processor core.

The present invention therefore calculates the utilization of eachprocessor/processor core, or rather any functional subsystem component,differently. The functionality provided herein provides that each of amultiplicity of “energy factors” may be calculated and measured inreal-time to obtain a total utilization of each processing component(e.g., in this example, processor/processor core) by comparing acalculated aggregate work of each of the energy factors against amaximum amount of power able to be consumed by the respectiveprocessor/processor core over a given period of time. That is, theutilization of each processor/processor core is not determined bywhether it “runs” processes or is “idle”, but rather takes intoconsideration “what” processes the processor/processor cores arerunning. The calculated energy factors may include an allocated voltageand clock speed of the processor/processor core (or an adjustment to theallocated voltage and clock speed), an amount of power being consumed(either in real-time or over a predetermined timeframe), a type ofprocess being performed (e.g. dependent upon code, SLA, or otherfactors), and/or the specific processor instructions being executed. Ofcourse, a multitude of other factors or subsets of factors may beconsidered as one skilled in the art would appreciate.

In addition, one needs to observe that in a processor chip (which has acollection of processor cores) that is limited to a maximum energy(power) dissipation based on the factors mentioned (e.g. powerdispassion/cooling design of the package, or power allocated to thatprocessor chip etc.), each of the cores may execute differentinstruction sets per their specific workloads, and/or may have differentcache memory hits (from perfect to bad) behavior which will stall themulti-issue memory transactions and require cores to wait for memoryreads (as core speed is much higher than memory latency read times). Assuch, each core may consume different energy levels. This can lead todifferent utilization of the energy provided to the processor chip as awhole (which has multiple cores each at different speeds and voltages).In attempt to use this energy over the processor chip, different coresmay have an opportunity to run at a higher voltage and higher clockspeeds (which is why a higher voltage is needed—to run at a higher clockspeed) so as to keep energy utilization as close to 100% as possible.

If a given processor chip (here again, the processor chip is used inthis example, where any functional component may be referenced) cannotuse its allocated energy, for whatever reason, and after attempting allthe processes provided herein to use as much energy as possible havingbeen allocated to that processor chip, it may be said the processor isunder-utilized if an overall utilization reading is determined to have anegative ratio. The overall utilization reading for each component orsubsystem (e.g. processor) may be calculated as: (actual used energy perchip−min energy per chip)/(max energy per chip−min energy per chip),where min energy per chip is energy the processor chip consumes even ifthe cores are not processing anything, max energy per chip is themaximum total energy the chip is allowed to consume, within the coolingdesign of the package, or, the allocated energy allowed for that chip,and actual used energy is the total energy per chip used. All this ismeasured over a time window T, which may be of short duration (e.g. afew seconds or a minute).

Various patterns of the overall utilization reading may be used toclassify workloads based upon a calculation of the overall utilizationreading for each component or subsystem over a predetermined period oftime. That is, the overall utilization readings for a processingsubsystem, a memory subsystem, an accelerator subsystem, and/or anetworking subsystem may be viewed as part of the power management ofthe disaggregated system over the predetermined time period to classifya detected similar incoming workload of the same class.

For example, the classifications may include: a) Cache friendlyworkloads which consume relatively little energy from memory subsystemand much more energy from a cache memory subsystem while core(s)subsystem and/or accelerators subsystem still consume energy forperforming a computation. b) Cache unfriendly workloads which consumerelatively more energy from the memory subsystem and much less energy bytheir cache memory subsystem while core(s) subsystem or acceleratorssubsystem still consumes energy for performing a computation. c)Computation-bound workloads which consume relatively more energy fromtheir cores or accelerators subsystems and very little from their memoryand/or their networking subsystems. d) Input/output (I/O) boundworkloads which consume relatively more energy from their networkingsubsystem and relatively little energy from their cores and acceleratorssubsystem.

These classifications may be used to dynamically allocate power to theindividual components or subsystems and/or dynamically allocate a numberof the individual components or subsystems to the detected incomingworkloads having similar past overall utilization readings. Consider anexample of a previous workload x, which, upon obtaining the overallutilization reading of all components or subsystems in the disaggregatedsystem has been found to be a computation-bound workload. When thisworkload x returns (whether the identical workload or a similarworkload), the disaggregated system is able to determine that workload xwill need more processor energy, or more processors allocated, toperform workload x efficiently. The disaggregated system also is able todetermine that workload x will not need as much memory and networkingpower, or a lesser number of memory and networking components allocated.Thus the disaggregated system is able to, in real-time, adjust either anumber of components allocated to individual workloads based upon theiroverall utilization readings from past experiences and/or adjust thepower allocated to individual components used to perform the workload(whether scaled-up or scaled-down). In other embodiments, this overallutilization reading pattern may be used to dynamically assign powerand/or components and subsystems to individual users, certainapplications, or a combination thereof.

Thus, calculating the aggregate work using the multiplicity of energyfactors and obtaining the total utilization of each of theprocessors/processor cores (or other components), may be used tore-allocate power between processors/processor cores or other subsystemsin order to balance power or otherwise de-allocate components orsubsystems as to obtain a same throughput using less energy throughoutthe system using the described algorithms below.

FIG. 3 illustrates a method 300 for measuring component utilization in acomputing system. The method 300 may be performed in accordance with thepresent invention in any of the environments depicted in FIGS. 1 and 2,among others, in various embodiments. Of course, more or less operationsthan those specifically described in FIG. 3 may be included in method300, as would be understood by one of skill in the art upon reading thepresent descriptions.

Each of the steps of the method 300 may be performed by any suitablecomponent of the operating environment. For example, in variousembodiments, the method 300 may be partially or entirely performed by aprocessor, or some other device having one or more processors therein.The processor, e.g., processing circuit(s), chip(s), and/or module(s)implemented in hardware and/or software, and preferably having at leastone hardware component may be utilized in any device to perform one ormore steps of the method 300. Illustrative processors include, but arenot limited to, a Central Processing Unit (CPU), a Graphical ProcessingUnit (GPU), an Application Specific Integrated Circuit (ASIC), a FieldProgrammable Gate Array (FPGA), etc., combinations thereof, or any othersuitable computing device known in the art.

The method 300 begins (step 302) by obtaining a server energyutilization reading of a statistical significant number of servers outof a total number of servers located in the datacenter by measuring, atpredetermined intervals, a collective energy consumed by all processingcomponents within each server (step 304). The collective energy ismeasured by virtually probing thereby monitoring an energy consumptionof individual ones of all the processing components to each collect anindividual energy utilization reading; wherein the individual energyutilization reading is aggregated over a predetermined time period tocollect an energy consumption pattern associated with the serverutilization reading (step 306). The method ends (step 308).

In some embodiments, the power management module 250 can prioritizeworkloads into different categories where each category includes rangesof allowed clock speed and voltage (e.g. High (clock speed range a-b);Medium (clock speed range c-d); Low (clock speed range e-f)). The powermanagement module 250 may further dynamically adjust the range of eachcategory (the voltage and the clock speed of the processors/processorcores) based on analysis of a usage pattern of the workloads andforecast of the clock speed requirements (e.g. some workloads may haveseasonal or periodic pattern of resource utilization). Of course, a hostof other categories and priorities may be used other than a “High,Medium, and Low” priority, category, or range depending on the actualimplementation, as one of ordinary skill in the art would appreciate.

Further actions may perform re-categorizing a workload based on itsusage requirements and forecast. For example, if a Low priority workloadalways requires a ceiling level of its clock speed range, then it shouldbe re-categorized as a Medium priority workload. Conversely, if a Highpriority workload generally requires a lower level of its allowed rangefor clock speed, then it may be re-categorized as a Medium priorityworkload, and so on. Additionally, power may be assigned to certainworkloads based on demand in order to increase performance (clock speed)of the processor/processor core by lowering the power and hence, clockspeed of other, lower priority or lower demand workloads. Furthermore,the mechanisms of the power management module 250 may determine a moreaccurate amount of contract power for a next term from aprediction/forecast of priority distribution.

Continuing to FIG. 4, a block diagram, high-level view 400 of oneembodiment of the present invention is illustrated. A historical usagedata 410 is analyzed by the system to estimate the clock speedfloor/ceiling range of the High, Medium, Low (or other predefined)categories 412. The system dynamically readjusts the range for High,Medium, Low categories based on the changes observed in the historicalusage data 410. In one example, the ranges for the categories may bestatically defined, whereas, in another example, the range may beautomatically estimated based on the historical usage pattern 410 andthe forecast of the demand.

The system uses the historical usage data 410 to categorize theworkloads in different priorities of clock speed and voltage 414. Aranking of workload groups 420 is performed based on SLA, priority,resource dependencies to determine the order in which the resources(power) are re-allocated 416. Different factors may be taken intoconsideration to form the criteria for grouping the workloads. Oneadvantageous set of criteria may be focused on workloads withcomplimentary resource usage patterns where resource movement will haveminimal impact. For example, a workload X with High SLA may have a highusage pattern and will be categorized in the High clock speed rangecategory versus another workload Y also has high SLA but may have a lowusage pattern will be categorized in the Low clock speed range category.Taking these workloads with complimentary resource usage pattern tore-allocate electrical power will help maximize the electrical powerutilization, hence minimizing the total available power while satisfyingthe SLA. Another set of criteria may be criteria in which the workloadowner specifies a group of workloads where there is a group performancegoal or group power savings goal in which the goals of the group aremore important than the goals of a particular workload. The system thendynamically re-allocates power to permit the higher clock speed from oneworkload to another to fulfill the clock speed demand 418.

Moreover, adjustments to the allocated or re-allocated voltage and clockspeed of each processor/processor core may be performed upon detectingan input electrical source (e.g. utility power) has been reduced. Forexample, during a “browning” of the utility power grid, an outrightpower failure requiring the need of Uninterruptible Power Supply (UPS)or generator usage, or other situations where the input power to thedatacenter is not at full capacity, power may be re-allocated from theprocessors (adjusting the voltage and clock speed therein) performinglower priority workloads to higher priority workloads, dependent uponSLA requirements of the respective workloads being performed.

Such is the case when the utility power is no longer provided, where adatacenter needs to first run for a time (e.g. a few seconds to aminute) on battery power (to allow proper shut down of workloads of SLAsthat have not been optioned to continue running during a power shortage)until backup generator(s) at the datacenter have been able to start andbring a certain amount of power online (i.e. the generator(s) maygenerate a portion of the total utility power used in normal operationonly to cover those workloads and SLAs having been contracted to run onsuch power interruption).

Generators take some time to power on and some time to change theiroutput power, in the same way a certain lag is seen when a car isaccelerated by pushing the gas paddle all the way to the floor beforefull acceleration is reached. Therefore, a UPS is often used at animmediate time after power interruption to enable the entire datacenterto get into a power emergency mode, shut down servers and services thatwhere not explicitly contracted for power shortage (e.g. by taking asnapshot of the memory image and pushing the image to flash memoryquickly to create a resume point later on). After a few seconds (e.g.10-30 seconds generally), the generator(s) are able to stabilize theirpower production to provide backup power to the rest of the services andresource pools that are contracted to keep running or are otherwiseengaged to run in case of a power shortage/emergency. During the backuppower generation, the aim is to save as much as possible the fuel neededto power the backup generator(s) and if a service or resource pool doesnot use or need the backup power, to shut it off to be later restartedwithin milliseconds based on the disaggregated architecture. Forservices/workloads/resource pools which are contracted to run on backuppower, the speed to execute the service would be the minimal possible tosustain the real-time throughput needed by that service and itsprocessing. Hence constant adjustments for clock speed and voltage areneeded to lower the energy consumption of that service/workload/resourcepool and only increase such when the throughput detected is slower thanthat needed for the real-time processing delivery as contracted.

Given that this processing can be erratic during the generation ofbackup power, the datacenter battery farm is used as a smoothingstatistical mean to mitigate the variation between the time taken toadjust the local generators power output and the time taken to have torespond to real events with additional energy needs for processing. Intimes of valley use, the extra power generated by the power generatorsis stored in the batteries until the generator(s) adjust its poweroutput to be lower (and consume less fuel) and vice versa, the batteryprovides the extra power needed momentarily before the generator cangenerate more power output and consume more fuel.

In some instances, depending on an amount of electrical power providedduring the reduction and/or interruption of the utility power by eitherthe UPS, the generator(s), or a combination thereof, power may bere-allocated (after a predetermined time period on generator power) fromone or more processors performing lower priority workloads (according toan SLA) to a higher priority workload, such that the lower priorityworkload is stopped or temporarily suspended until normal utility poweris restored.

In other instances, certain individual components or subsystems (i.e.processor cores, processors, etc.), regions of subsystems (i.e. pools ofprocessors, pools of memory components, etc.), or entire composeddisaggregated systems may not be provided with backup electrical power,given the components, regions, or disaggregated systems are executingworkloads having SLA requirements that do not include backup powerduring an interruption. For example, during a utility power interruption(or upon the detection of a utility power reduction over a predeterminedthreshold), a UPS in the datacenter may keep certain components orsystems supplied with power until generator(s) are started and beginsupplying power to the datacenter, including the individual componentsor systems. After a predetermined time period of running on generatorpower, the systems may start to systematically shut down (and thereforeshut off power) to the individual components or systems. This mayinclude disaggregated systems shutting down granular components such asindividual processors or processor cores, pools of subsystems (e.g.pools of processors, memory components, etc.) or may include shuttingdown one or more entirely composed disaggregated systems.

In another example, certain “regions” of the datacenter may bedesignated for running workloads having SLAs of varying priority. Oneregion of the datacenter (having multiple disaggregated systems therein)may be designated for running high priority SLA workloads, one regionmay be designated for running medium priority SLA workloads, and oneregion may be designated for running low priority SLA workloads. Duringthe utility power reduction or interruption, the region of thedatacenter (and thus the multiple disaggregated systems therein) runningthe low priority SLA workloads may be shut down and removed from thegenerator power after the predetermined time period has elapsed. To thisend, the entire region may be shut down and removed from generator powersimultaneously, or certain disaggregated systems within the region mayfirst be shut down, followed by other systems, followed by more systems,in a systemic process according to an SLA hierarchy of the workloadsbeing performed. Of course, as mentioned herein, the “high priority”,“medium priority”, and “low priority” is only representative of a fewexamples to clearly explain the invention. In an actual implementation,a variety of priority levels, SLA considerations, or other factors maybe considered according to the specific needs of the implementation.

These workloads/SLAs can be of many use cases. For example, thedatacenter may have workloads that are in an alternate site for aprimary datacenter in case of disaster recovery, that usually do not runbut rather are maintained in an updated state from a first datacenter,but may be required to run if a disaster impacts the primary (first)datacenter location. Another use case may consist of workloads ofgenerally high priority SLAs which have an exception not to run in sucha disaster recovery or power outage situation.

Thus, if a total power consumption of a datacenter is say 10 Megawattsduring normal operation, it may only need 1 Megawatt of emergency powergeneration which ultimately saves costs for the datacenter provider andthe user. Using the power balancing algorithms and methodologiesprovided herein, some high priority SLAs may not actually consume allthe power as they normally would, as they either may do less work and/orsuch can be run at lower clock speeds and voltages so as to conserve thebackup power needed.

Moreover, using these models and methods, overall power consumption canbe estimated based usage pattern and forecast for a next term tocontract a more accurate amount of electrical power to meet the workloaddemands of the datacenter 424.

Advancing, another embodiment is disclosed with reference to FIG. 5.FIG. 5 illustrates a method 500 for power management in disaggregatedcomputing systems, in accordance with one embodiment of the presentinvention. The method 500 may be performed in accordance with thepresent invention in any of the environments depicted in FIGS. 1 and 2,among others, in various embodiments. Of course, more or less operationsthan those specifically described in FIG. 5 may be included in method500, as would be understood by one of skill in the art upon reading thepresent descriptions.

Each of the steps of the method 500 may be performed by any suitablecomponent of the operating environment. For example, in variousembodiments, the method 500 may be partially or entirely performed by aprocessor, or some other device having one or more processors therein.The processor, e.g., processing circuit(s), chip(s), and/or module(s)implemented in hardware and/or software, and preferably having at leastone hardware component may be utilized in any device to perform one ormore steps of the method 500. Illustrative processors include, but arenot limited to, a Central Processing Unit (CPU), an Application SpecificIntegrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), etc.,combinations thereof, or any other suitable computing device known inthe art.

Beginning (step 502), A usage pattern is modeled for each workload usingwell-known methods such as standard regression modeling techniques (step504). As discussed, as small variation in power utilization ismaintained within the contracted electric power range, where the goal isto drive the Total_Available close to 0 (such that maximum utilityprovided power is used), however, under all circumstances theTotal_Available should be less than the Total_Contracted timesContracted_Variance. In a traditional server bound environment, however,it may be desirable to maintain a power reserve to quickly allocate tohigh priority SLAs if erratic needs arise but in a disaggregatedenvironment, re-allocation of resources may be triggered quickly withinmilliseconds.

Continuing, within each Time_Window t (step 506), a power managementalgorithm 550 is calculated as follows:

For each workload i (step 510), the Workload_Resource_Overage_(i) iscomputed and stored in the workload statistics database 254 (step 512),expressed as:Workload_Resource_Overage_(i)=Workload_Current_Allocated_(i)−Workload_Predicted_Demand_(i)

The Workload_Resource_Overage is ordered, for example, based on an SLAand/or an impact policy, and/or a business logic (step 514). Adetermination is made as to whether the workload i has aWorkload_Resource_Overage>0 or a Workload_Resource_Overage<0 (step 516).If workload i has a Workload_Resource_Overage>0, resources are removedfrom all workloads with positive Workload_Resource_Overage_(i) therebyhaving a surplus of resources (step 518), and the Total_Available iscomputed (step 520), expressed as:

-   -   For each workload i:        If Workload_Resource_Overage_(i)>0 then,        Workload_Current_Allocated_(i)=Workload_Current_Allocated_(i)−Resource_Unit        Total_Available=Total_Available+Resource_Unit

If, at step 516, the workload i has a Workload_Resource_Overage<0,resources are allocated to workloads having a negativeWorkload_Resource_Overage thereby having a greater demand for resources(step 522), and the Total_Available is computed (step 524), expressedas:

-   -   For each workload i:        If Workload_Resource_Overage_(i)<0 then,        Workload_Current_Allocated_(i)=Workload_Current_Allocated_(i)+Resource_Unit        Total_Available=Total_Available−Resource_Unit

Continuing, a determination is then made as to whether theTotal_Available>0 (or >>0) or the Total_Available<0 (step 526). If, atstep 526, the Total_Available>0 (or >>0), resources are proportionallydistributed to the existing workload or additional workloads that areable to run, based on the proportion of the existing allocation, ormanual input regarding workload expansion as based on the specifiedpolicy and the SLA (step 528), expressed as:

-   -   For each workload i,        Workload_Allocated_Proportion_(i)=Workload_Current_Allocated_(i)/Total_Workload_Allocated        Workload_Current_Allocated_(i)+=Workload_Allocated_Proportion_(i)*Total_Available        Total_Available=0

If, at step 526, the Total_Available is below 0, then resources areproportionally removed from the Workload_Current_Allocated_(i) or theworkload may be suspended, based on the proportion of the existingallocation or manual input regarding workload expansion in the SLA (step530), expressed as:

-   -   For each workload i,        Workload_Allocated_Proportion_(i)=Workload_Current_Allocated_(i)/Total_Workload_Allocated        Workload_Current_Allocated_(i)−=Workload_Allocated_Proportion_(i)*Total_Available        Total_Available=0

For each workload i, the model referenced in step 504 is iterativelyupdated using the power management algorithm 550.

In some embodiments, a priority order may be used to allocate additionalpower between workloads when the Total_Available>0, as illustratedwithin the chart in FIG. 6. Table 600 shows an example of a priorityorder that may be used to allocate the additional power betweenworkloads when total available power is above zero, in order of priority1 to 4. Table 600 illustrates that a first priority may be based on acritical, predictable SLA, a second priority based on an erratic, yetcritical SLA, a third priority based on an opportunistic and predictableSLA, and a fourth priority based on an erratic, opportunistic SLAworkload. The same order applies in reverse for determining priority forremoving power when the Total_Available is below 0 (i.e. from workloadpriorities 4 to 1). Of course, as one of ordinary skill in the art wouldappreciate, there may be additional SLA priorities beyond thosedescribed which will modify the above order of allocating power in thesame proportion.

In other embodiments, the present invention provides for novelmechanisms to measure a total utilization of entire datacenters, byaggregating energy measurements provided from processing components asgranular as a processor core (i.e. a portion of a processor), to rackscomprising multiple processors, up to and including all components (e.g.accelerators, networking components, memory, cache etc.) within theentire datacenter itself. Instead of collecting this measurement from a“power room” in the datacenter, this methodology of energy measurementprovides the unique opportunity for individuals who do not either haveaccess or whom are otherwise more casual users (not high-leveladministrators) to obtain metrics regarding an actual utilization of thecomponents within the datacenter in order to achieve a better overallefficiency and throughput of such components, and thus allow for usersto consider the collection of the resources they have rented and howthey can optimize such, in addition to the datacenter operators whom canover-provision resources within the different SLA priorities and stillperform to the various SLAs conditions with additional revenue generatedon same cost platform. These mechanisms are in contrast to obtainingtemperature or other complicated and intrusive readings to determineresource and/or datacenter utilization as commonly applied in prior art.

For example, when a user whom is not the datacenter owner has a need toknow what the utilization of the datacenter is (as defined above byenergy based utilization and not operating system based state ofprocesses utilization), virtual probes may be implemented to read energyregisters of processing components, or portions thereof (processorcores, etc.), over multiple different servers. Such a probe may beincorporated as the smallest possible Virtual Machine (VM) entity thatcan be defined at a cloud service provider.

There are two aspects of this probe. A probe normally is reading, atcertain periods or intervals of time (e.g. a minute), the energyconsumption register of the processor/server (which aggregates allcores' energy use for that server). Such may have also readings of theprocessor package energy, the cores energy, the memory energy, theaccelerators' (e.g. GPU cores or FPGAs if available) energy, and thenetworking components' energy.

In probe mode, such a small VM runs over a long period (week, month orso) and collects the following data: the maximum energy ever observed inthe processor; the minimum energy ever observed in the processor; andthe average of energy observed over time, by calculating energy readingdifferences read each time the probe was reading such for the processorchip/server. If a probe happens to be in a server that always is busy,it will always read the maximum energy. Similarly, if a probe happens tobe in an idle server, it will always read the minimum energy.

Next, over all the probes running in the datacenter, a global minimumand a global maximum energy is calculated for those probes allocated inthe same type of a server (where the server type references to differentspecifications that may be possible for servers provided by a cloudservice operator—for example, in a different type the server may havedifferent processor versions or use older generations of processors vs.newer generations, etc.). For all probes allocated to the same type ofservers, a global maximum and minimum are calculated at a predeterminedtime to obtain the overall utilization of the datacenter among allprobes incorporated into servers of same type.

Provided that there are a statistically meaningful number of probesmonitoring different servers each (as discussed below), at thedatacenter, an overall utilization of that datacenter may be calculatedas energy or power utilization to be: (average of all probes (or averageenergy seen over time by each)−global minimum power calculated from allprobed servers)/(global max power calculated from all probedservers−global minimum power calculated from all probed servers).

To add new probes to an existing set of probes, each of the existing setof probes run a “power virus” (a set of instructions designedspecifically for generating a maximum energy consumption per the tiny VMallocated in a server). The run/stop time pattern of the power viruswill be a key to be detected by the new candidate monitoring probe ofthe VM being allocated. The power virus in all existing VM probes isthen run at times that can be detected as such (e.g. off 10 seconds, on20 seconds, off 20 seconds, on 5 seconds, and more), to form apeak/valley chain of detectable changes in energy by the new candidateprobe in a new tiny VM. This pattern or key repeats over a longer periodof time (e.g. minutes to an hour or so) to allow accurate detection bythe new candidate probe.

The new candidate probe then reads the energy the processor chip orserver is using as it is able to observe. If there is any correlationwith the power virus patterns of on/off periods the other (existing)probes generate, the new candidate probe will be released and a randomwaiting time will be waited before a new VM be allocated in which theprocess begins again. Similarly, if the new candidate probe continuallyobserves a maximum energy output with no changes, the new candidateprobe will be released and after a random waiting time, a new probe willbe provisioned with a new VM allocation. Thus the process to add newprobes will ensure that only unique probes running on different serverswere allocated before turning all of them to be probing of energy/powerconsumption.

The allocation of new probes will continue and can happen over few daysor more, until a statistically meaningful number of probes based on theknown or estimate size of a datacenter location where a cloud service isrunning, is successfully allocated. This number is derived from knownart of statistics. Moreover, the total time of monitoring the datacenterin this way should also be long enough to observe most of the changesdue to day of the week or mask out other events (e.g. holidays, vacationtimes in summer or starting a new datacenter vs established one) whichmay affect the average utilization measured.

In other embodiments, on the user measured side, a company that isrunning hundreds of instances of servers composed out of resources inthe disaggregated system (where “compose” means a system is put togetherby wiring, in this example with optical switches connecting fibers toform end-to-end connections between subsystems or resource pools), mayalso have direct access to read energy consumption of all allocatedservers and thus may also evaluate workload performance and howefficient their use is. This can benefit users to perhaps reduce cost atleast at some times, depending on the pattern of energy use over alltheir allocated resources and the subsystems/resource types used and atwhat times, etc. This functionality is directly attributed to the use ofdisaggregated systems. Note, again, this disaggregation is separate fromon-demand model that cloud computing generally provides. In theon-demand model, users may acquire more resources (VMs) up to a limit ofthe pre-built physical server boxes that ultimately host a number ofusers' VMs. In the disaggregated design, the resources are attached peruser as needed, and in a more elastic format since the pools ofresources are very large and thus adding processors or other components(e.g. scale up or SMP with multiple sockets) can be formed as neededwithout the limitation of whether or not the resources physically residein the same server “box”.

Conversely, the owner or operator of the datacenter will read energyconsumption for all composed servers, and the resources in question(e.g., processors, memory, accelerators, storage, communicationnetworking, cache in each processor, and any/all functional subsystemsof interest). Such access to read the energy consumption for allsubsystems can be facilitated through the hypervisor or the controlnetwork the datacenters' operators have access to as part of the overallmanagement of the datacenter.

The aggregated readings of energy per a period of time will give theoperators an accurate state of usage and profile of operation for allworkloads running in the datacenter. Such may be in terms of processorsutilizing energy. For example, if processors are constantly running at alow percent of utilization, the operator may add more workloads to beallocated or may reduce number of processors in the processor pools usedfor the datacenter, etc.

This methodology applies as a hierarchy of consideration as well. Forexample, some of the “regions” in the datacenter may be more utilizedthan others. The regions are areas that resource pools can be shared inthe dynamic composition of servers. Other regions may be less utilized,hence balancing energy and workloads between regions may offer betterutilization and provide better efficiency over time. This can also bevisible from memory energy use versus cache or versus accelerators(GPUs, FPGAs, etc.) which all help to better decide matching ofresources in regions and/or all of the datacenter that is measured.

It is important to note these mechanisms are provided in addition to thediscussed methods and schemes a user can perform without having theaccess as a datacenter operator (where the user accesses energyconsumption via probe/power virus VMs etc.). Hence, as a datacenteroperator, there is no need to use probe or power virus VMs since theoperator can read directly any energy consumption at any location andany subsystem/functional unit and such operator clearly can address andthereby knows the location and IDs of all associated resources withoutdoing any statistical meaningful probing as is the case of thenon-operator.

The present invention may be an apparatus, a system, a method, and/or acomputer program product. The computer program product may include acomputer readable storage medium (or media) having computer readableprogram instructions thereon for causing a processor to carry outaspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

What is claimed is:
 1. A method for measuring component energyutilization in a datacenter, by a processor, comprising: obtaining aserver energy utilization reading of a statistical significant number ofservers out of a total number of servers located in the datacenter bymeasuring, at predetermined intervals, a collective energy consumed byall processing components within each server; wherein the server energyutilization reading is used to dynamically allocate electrical power torespective processing components within each server; measuring thecollective energy by virtually probing thereby monitoring an energyconsumption of individual ones of all the processing components to eachcollect an individual energy utilization reading; wherein the individualenergy utilization reading is aggregated over a predetermined timeperiod to collect an energy consumption pattern associated with theserver energy utilization reading; and wherein the energy consumptionpattern comprises observed energy consumption values of a maximumconsumption, a minimum consumption, and an average consumption betweenthe maximum consumption and the minimum consumption of the server energyutilization reading; calculating a global minimum consumption value of aplurality of servers having equal specifications by measuring a minimumobserved energy or an average minimum observed energy detected over aplurality of virtual probes monitoring the processing components withineach of the plurality of servers; calculating a global maximumconsumption value of the plurality of servers having equalspecifications by measuring a maximum observed energy or an averagemaximum observed energy detected over the plurality of virtual probesmonitoring the processing components within each of the plurality ofservers; allocating a new virtual probe by running a power virus onpreviously allocated ones of the plurality of virtual probes to generatefor each an energy consumption key, the energy consumption key used toform a peak and valley chain of detectible energy changes to compareagainst the energy consumption key of a candidate virtual probe; whenthe energy consumption key of the candidate virtual probe matches theenergy consumption key of a previously allocated one of the plurality ofvirtual probes, releasing the candidate virtual probe and creating awaiting time before allocating an alternative candidate virtual probe;and attempting to allocate the new virtual probe until the server energyutilization reading of the statistical significant number of servers isreached.
 2. The method of claim 1, wherein the processing componentscomprise at least portions of processors, processor cores, memorycomponents, internal or external accelerators, and networkingcomponents.
 3. The method of claim 1, wherein virtually probingcomprises running a process on a Virtual Machine (VM) while reading anenergy register of the processing components; and further includingobtaining a total utilization of the datacenter according to thealgorithm: (average of all probes (or average energy seen over time byeach)−global minimum power calculated from all probed servers)/(globalmaximum power calculated from all probed servers−global minimum powercalculated from all probed servers).
 4. The method of claim 1, furtherincluding presenting the server energy utilization reading to anoperator of the datacenter, whereby the operator uses the serverutilization reading to adjust at least one of a number of workloadsbeing performed and a number of the processing components allocated toperform respective workloads.
 5. A system for measuring component energyutilization in a computing system, the system comprising: at least oneprocessor device, wherein the at least one processor device: obtains aserver energy utilization reading of a statistical significant number ofservers out of a total number of servers located in the datacenter bymeasuring, at predetermined intervals, a collective energy consumed byall processing components within each server; wherein the server energyutilization reading is used to dynamically allocate electrical power torespective processing components within each server; measures thecollective energy by virtually probing thereby monitoring an energyconsumption of individual ones of all the processing components to eachcollect an individual energy utilization reading; wherein the individualenergy utilization reading is aggregated over a predetermined timeperiod to collect an energy consumption pattern associated with theserver energy utilization reading; and wherein the energy consumptionpattern comprises observed energy consumption values of a maximumconsumption, a minimum consumption, and an average consumption betweenthe maximum consumption and the minimum consumption of the server energyutilization reading; calculates a global minimum consumption value of aplurality of servers having equal specifications by measuring a minimumobserved energy or an average minimum observed energy detected over aplurality of virtual probes monitoring the processing components withineach of the plurality of servers; calculates a global maximumconsumption value of the plurality of servers having equalspecifications by measuring a maximum observed energy or an averagemaximum observed energy detected over the plurality of virtual probesmonitoring the processing components within each of the plurality ofservers; allocates a new virtual probe by running a power virus onpreviously allocated ones of the plurality of virtual probes to generatefor each an energy consumption key, the energy consumption key used toform a peak and valley chain of detectible energy changes to compareagainst the energy consumption key of a candidate virtual probe; whenthe energy consumption key of the candidate virtual probe matches theenergy consumption key of a previously allocated one of the plurality ofvirtual probes, releases the candidate virtual probe and creating awaiting time before allocating an alternative candidate virtual probe;and attempts to allocate the new virtual probe until the server energyutilization reading of the statistical significant number of servers isreached.
 6. The system of claim 5, wherein the processing componentscomprise at least portions of processors, processor cores, memorycomponents, internal or external accelerators, and networkingcomponents.
 7. The system of claim 5, wherein virtually probingcomprises running a process on a Virtual Machine (VM) while reading anenergy register of the processing components; and wherein the at leastone processor device obtains a total utilization of the datacenteraccording to the algorithm: (average of all probes (or average energyseen over time by each)−global minimum power calculated from all probedservers)/(global maximum power calculated from all probed servers−globalminimum power calculated from all probed servers).
 8. The system ofclaim 5, wherein the at least one processor device presents the serverutilization reading to an operator of the datacenter, whereby theoperator uses the server energy utilization reading to adjust at leastone of a number of workloads being performed and a number of theprocessing components allocated to perform respective workloads.
 9. Acomputer program product for measuring component energy utilization in acomputing system, by a processor device, the computer program productembodied on a non-transitory computer-readable storage medium havingcomputer-readable program code portions stored therein, thecomputer-readable program code portions comprising: an executableportion that obtains a server energy utilization reading of astatistical significant number of servers out of a total number ofservers located in the datacenter by measuring, at predeterminedintervals, a collective energy consumed by all processing componentswithin each server; wherein the server energy utilization reading isused to dynamically allocate electrical power to respective processingcomponents within each server; an executable portion that measures thecollective energy by virtually probing thereby monitoring an energyconsumption of individual ones of all the processing components to eachcollect an individual energy utilization reading; wherein the individualenergy utilization reading is aggregated over a predetermined timeperiod to collect an energy consumption pattern associated with theserver energy utilization reading; and wherein the energy consumptionpattern comprises observed energy consumption values of a maximumconsumption, a minimum consumption, and an average consumption betweenthe maximum consumption and the minimum consumption of the server energyutilization reading; an executable portion that calculates a globalminimum consumption value of a plurality of servers having equalspecifications by measuring a minimum observed energy or an averageminimum observed energy detected over a plurality of virtual probesmonitoring the processing components within each of the plurality ofservers; an executable portion that calculates a global maximumconsumption value of the plurality of servers having equalspecifications by measuring a maximum observed energy or an averagemaximum observed energy detected over the plurality of virtual probesmonitoring the processing components within each of the plurality ofservers; an executable portion that allocates a new virtual probe byrunning a power virus on previously allocated ones of the plurality ofvirtual probes to generate for each an energy consumption key, theenergy consumption key used to form a peak and valley chain ofdetectible energy changes to compare against the energy consumption keyof a candidate virtual probe; an executable portion that, when theenergy consumption key of the candidate virtual probe matches the energyconsumption key of a previously allocated one of the plurality ofvirtual probes, releases the candidate virtual probe and creating awaiting time before allocating an alternative candidate virtual probe;and an executable portion that attempts to allocate the new virtualprobe until the server energy utilization reading of the statisticalsignificant number of servers is reached.
 10. The computer programproduct of claim 9, wherein the processing components comprise at leastportions of processors, processor cores, memory components, internal orexternal accelerators, and networking components; and further includingan executable portion that obtains a total utilization of the datacenteraccording to the algorithm: (average of all probes (or average energyseen over time by each)−global minimum power calculated from all probedservers)/(global maximum power calculated from all probed servers−globalminimum power calculated from all probed servers).
 11. The computerprogram product of claim 9, wherein virtually probing comprises runninga process on a Virtual Machine (VM) while reading an energy register ofthe processing components.
 12. The computer program product of claim 9,further including an executable portion that presents the serverutilization reading to an operator of the datacenter, whereby theoperator uses the server energy utilization reading to adjust at leastone of a number of workloads being performed and a number of theprocessing components allocated to perform respective workloads.